A typical conventional SIEM system is fully centralized. The centralized SIEM system collects raw log information from monitored remote applications of an enterprise environment, and uses the collected raw log information to build a comprehensive database of application activity. The system subsequently performs correlations on the data stored in the database to determine, for example, if specified patterns are found.
This conventional centralized approach has a number of significant drawbacks. For example, collecting and indexing raw log information in a centralized location increases latency. Also, many of the desired correlations may not require the complete environment context, and so working in a single large centralized database slows the system performance, as much of the data collected and indexed is not relevant to a particular query. In addition, since there are usually many queries to be correlated, and these are sharing a single database, it is unduly complex to prioritize and otherwise schedule resources for a selected subset of queries that might affect a subset of users or services. These and other problems create serious scalability issues for centralized SIEM systems, and as a result it is becoming increasingly difficult to implement such systems in large-scale public or private clouds or using other types of distributed virtual infrastructure.
Another important drawback of the centralized SIEM approach is loss of application context. Since the log information is transmitted to a single collection point, relevant application context may be lost. For example, the log information may not contain all of the disk or memory program context that existed at the moment the log record was persisted. By its nature, a log record is a very specific, limited summary of something that the application chooses to record. It is not everything that is known to the application when the log was written. More information, such as current power consumption, or the identification of other processes running on the host, may be relevant, but will not be logged natively by the application, and therefore will not be accessible to the centralized SIEM system.